Analysis-by-synthesis in Auditory-visual Speech Perception: Multi-sensory Motor Interfacing

In conversational settings, one sees as much as one hears the interlocutor. Compelling demonstrations of auditory-visual integration in speech perception are the classic McGurk effects [1]: in McGurk “fusion,” an auditory [p] dubbed onto a face articulating [k] is perceived as a single fused percept [t], but in McGurk “combination,” an auditory [k] dubbed onto a visual [p] is heard as multiple combinations of [k] and [p]. The natural spatio-temporal co-occurrence of auditory-visual (AV) speech signals is thus a likely feature used by the brain. AV speech integration offers interesting challenges for neuroscience and speech science alike. How, when, where, and in what format do auditory and visual speech signals integrate? A set of empirical studies are described, whose results suggest that multisensory speech integration relies on a dynamic set of predictive computations involving a large-scale cortical network (including sensory and motor systems). In building on the classical ‘analysis-by-synthesis’ framework [2], it is suggested that speech perception entails a predictive brain network, which computationally operates on abstract speech units.